Chatterbox Challenge as a Test-Bed for Synthetic Emotions

Chatterbox Challenge is an annual web-based contest for artificial conversational systems, ACE. The 2010 instantiation was the tenth consecutive contest held between March and June in the 60th year following the publication of Alan Turing’s influential disquisition ‘computing machinery and intelligence’. Loosely based on Turing’s viva voca interrogator-hidden witness imitation game, a thought experiment to ascertain a machine’s capacity to respond satisfactorily to unrestricted questions, the contest provides a platform for technology comparison and evaluation. This paper provides an insight into emotion content in the entries since the 2005 Chatterbox Challenge. The authors find that synthetic textual systems, none of which are backed by academic or industry funding, are, on the whole and more than half a century since Weizenbaum’s natural language understanding experiment, little further than Eliza in terms of expressing emotion in dialogue. This may be a failure on the part of the academic AI community for ignoring the Turing test as an engineering challenge. it” (1950, section 6, p. 445), Turing countered showing this stance was a solipsistic one. To say that “no mechanism ... could feel pleasure at its successes, grief when its valves fuse, be warmed by flattery, be made miserable by its mistakes, be charmed by sex, be angry or depressed when it cannot get what it wants” (p. 446), was, according to Turing, an extreme position: “the only way by which one could be sure that a machine thinks is to be the machine and to feel oneself thinking” Turing’s emphasis (ibid). Turing replied, as Stins and Laureys put it “in a succinct British fashion” (2009, p. 265) that rather than labouring over the point “A is liable to believe A thinks but B does not while DOI: 10.4018/jse.2010070102 International Journal of Synthetic Emotions, 1(2), 12-37, July-December 2010 13 Copyright © 2010, IGI Global. Copying or distributing in print or electronic forms without written permission of IGI Global is prohibited. B believes B thinks but A does not” it is “usual to have the polite convention that everyone thinks” (1950, p. 446). There are a variety of objections to the “Turing test” and whether it can really be a way to assert whether a computer is thinking or not, it is beyond the scope of this paper to review them, readers are directed to Shah and Warwick (2010b) nevertheless the concept of being able to pass the Turing test is a complex endeavour indeed, and analysing such a variant Turing test contest can be of great philosophical value. Turing put forward possible questions and answers to show that if the responses were “satisfactory and sustained” then one might not describe the answers as “an easy contrivance” (1950, p. 447). It is this method to assess whether machine responses to questions are satisfactory and sustained that is evaluated in an annual contest – the Chatterbox Challenge. Artificial conversation systems (ACE), commonly known as ‘chatbots’ compete against each other across a number of categories. What is considered a satisfactory response can be subjective; one interrogator may find a response inappropriate while another may accept it as humorous. An answer to a question may seem ersatz-like, but may also be a satisfactory emotive response under interrogation or conveying disinterest in topic by the ‘witness’. Whether machines are capable of expressing emotion through their responses or whether they are still Eliza-like (Weizenbaum, 1966) can be found by analysing the contest’s transcripts. Emotion: mood, feeling or Expression? According to Broekens “Emotion is a complex topic, and agreement on one solid definition does not really exist” (2010). What does stand as an ‘emotion’ is also not clear cut (Barrett et al., 2007), and more if we take into account the qualia aspects of emotions, their feeling, we need to paraphrase Augustine of Hippo1 ideas on his subjective theory of time telling that “What then are emotions? If no one asks me, I know: if I wish to explain it to one that asked, I know not”. In previous studies by two of the current authors (Vallverdú & Casacuberta, 2008, 2009), some neurological, philosophical and computational approaches to the analysis of natural or synthetic emotions were described. It is beyond the scope of the present paper to explain again all this information, but adding some basic different ideas such as Llinás (2001) and Damasio (1999), we provide a more sophisticated approach to the meaning of emotion from an evolutionary perspective. From our new and original point of view, emotions are: 1. Embodied-intentionality generators: with a genotypic punishment/reward management system pre-encoded in our bodies, emotions transforms bodies into goaloriented complex systems. Basically, they are: feeding, self-preservation and reproduction (all these activities involve movement). Beings are not just things in the world, but things with an evolutionary force embedded in them. Under this embodied frame, we are emotionally oriented toward certain events of the world. In a certain way, emotions emerge from an embodied neurological a priori focus for the logic of survival. Interpreting in a different way the existentialist motto “existence precedes essence”, we consider that the essence (or main intentionality) of human beings is wired into their physical structure (the body). In this sense, and avoiding a defence of the naturalist fallacy, we act in pre-organized ways because we have been programmed for specific reactions to basic events (at least those for which our evolutionary ancestors were faced to through their lives). Under this new light, essence and existence are equivalent concepts. Only cultural information can modify to some extent the internal rules of our bodies. For all the previous reasons, we would wish that the results of neurophenomenology were included into the daily agenda of (natural or artificial) mind researchers. 24 more pages are available in the full version of this document, which may be purchased using the "Add to Cart" button on the publisher's webpage: www.igi-global.com/article/chatterbox-challenge-test-bed-